Computing the need for standardization of a set of values

ABSTRACT

A method, system and computer program product for determining a data standardization score for an attribute of a dataset. A data standardization score is calculated, which reflects whether data quality of attribute values would increase if a standardization rule is applied to the attribute values. Based on attribute metadata, it may be determined whether an indication to carry or not to carry out standardization is available for at least part of the attribute values of the dataset. In response to finding the indication, a respective value may be set for the data standardization score. In response to not finding the indication, a data standardization score algorithm may be run on the at least part of the attribute values of the dataset. The data standardization score value may be compared to a predefined criterion to determine whether data standardization is to be applied on the attribute.

TECHNICAL FIELD

The present invention relates generally to digital computer systems, andmore particularly to determining a data standardization score for anattribute of a dataset.

BACKGROUND

Data quality improvement is achieved through data cleansing whichtypically has four stages namely, investigate, standardize,de-duplication, and survivorship. In the standardization, stage data istransformed to a standard uniform format. This involves segmenting thedata, canonicalization, correcting spelling errors, enrichment, andother cleansing tasks using rule sets. Different rule sets need to becreated for data from different domains. However, creation of datastandardization rules is an expensive task and can easily take months ofeffort if not weeks. Thus, there is a need to control the creation ofthe data standardization rules.

SUMMARY

In one embodiment of the present invention, a computer implementedmethod for determining a data standardization score for an attribute ofa dataset comprises providing attribute metadata descriptive of theattribute. The method further comprises providing a data standardizationscore algorithm for finding potential duplicates in attribute values andcalculating a data standardization score accordingly, the calculateddata standardization score reflecting whether data quality of attributevalues would increase if a standardization rule is applied to theattribute values. The method additionally comprises determining, basedon the metadata for the attribute, whether an indication to carry or notto carry out standardization is available for at least part of attributevalues of the dataset. Furthermore, the method comprises in response tofinding the indication, setting a respective value for the datastandardization score. Additionally, the method comprises in response tonot finding the indication, running the data standardization scorealgorithm on the at least part of attribute values of the dataset. Inaddition, the method comprises comparing, by a processor, the datastandardization score value to a predefined criterion to determinewhether data standardization is to be applied on the attribute.

Other forms of the embodiment of the method described above are in asystem and in a computer program product.

The foregoing has outlined rather generally the features and technicaladvantages of one or more embodiments of the present invention in orderthat the detailed description of the present invention that follows maybe better understood. Additional features and advantages of the presentinvention will be described hereinafter which may form the subject ofthe claims of the present invention.

BRIEF DESCRIPTION OF THE DRAWINGS

A better understanding of the present invention can be obtained when thefollowing detailed description is considered in conjunction with thefollowing drawings, in which:

FIG. 1 represents a computerized system, suited for implementing one ormore method steps as involved in the present disclosure;

FIG. 2 is a flowchart of a method for determining a data standardizationscore for an attribute of a dataset in accordance with an embodiment ofthe present invention;

FIG. 3 is a flowchart of a method for finding groups of potentialduplicates using a standardization scoring algorithm in accordance withan embodiment of the present invention;

FIG. 4 is a flowchart of a method for determining whether an indicationto carry or not to carry out standardization is available in accordancewith an embodiment of the present invention;

FIGS. 5A-5D are diagrams depicting a method for creating clusters ofattribute values at various stages of processing using a bitmap inaccordance with an embodiment of the present invention; and

FIGS. 6A-6B depict an example format in which information may bepresented in accordance with an embodiment of the present invention.

DETAILED DESCRIPTION

The descriptions of the various embodiments of the present inventionwill be presented for purposes of illustration, but are not intended tobe exhaustive or limited to the embodiments disclosed. Manymodifications and variations will be apparent to those of ordinary skillin the art without departing from the scope and spirit of the describedembodiments. The terminology used herein was chosen to best explain theprinciples of the embodiments, the practical application or technicalimprovement over technologies found in the marketplace, or to enableothers of ordinary skill in the art to understand the embodimentsdisclosed herein.

The standardization refers to a process of transforming data to apredefined data format. The data format may include a common datadefinition, format, representation and structure. The data that is to betransformed is the data that is not to conform to the predefined dataformat. For example, the process of transforming the data may compriseprocessing the data to automatically transform the data where necessaryto comply with those common representations that define the data format.This process of transforming data may include identifying and correctinginvalid values, standardizing spelling formats and abbreviations, andvalidating the format and content of the data.

The present method may identify potential duplicates, wherein two valuesare potential duplicates if the two values are identical duplicates orthe two values are not identical yet but they may represent the sameentity. For example, the two values may be duplicates (e.g., representthe sane entity or the same attribute value) but may have spellingerrors and/or may have a respective order of their constituent partswhich is different. For example, the two values “Toyota Camery” and“Camry Toyota”.

Rather than depending on good feeling or experience in other domains,the present method provides data scientists with a metric to identifywhether standardization is required for a particular column. This metricmay become more important if the end user is a non-technical or in somecases another software program. The present method may be particularlyadvantageous for big data as the generation of metrics may be scalablefor large volumes.

For example, during data profiling, a score may be generated for eachcolumn that will indicate how “dirty” the column is in terms ofstandardization. In another words, a metric is generated that willindicate the confidence that the quality of data would improve if astandardization process would be applied on them.

In absence of such a method a data scientist would look at sample data,identify manually columns that could benefit from standardization andtest available standardization rules. The present method mayautomatically generate a score to gauge the need for standardizing aparticular column of data. The present method may scale in a big dataenvironment or in a data reservoir.

According to one embodiment, the attribute values are all attributevalues of the attribute in the dataset. This may be advantageous as itmay enable an accurate scoring method based on the full dataset. This isin contrast to a method using a sample of the dataset, where the resultsmay not be reliable.

According to one embodiment, the method further comprises providing aset of criterions. The determining, based on the metadata for theattribute, whether an indication to carry or not to carry outstandardization is available comprises: checking each of the criterionsfor the values of the attribute. This embodiment may prevent creationand running of standardization score algorithm when other conditions canbe easily checked for deciding to do or not to do the standardization.The creation and running of standardization score algorithm may be veryexpensive in term or processing resources compared to the checking ofthe set of criterions based on metadata.

According to one embodiment, the set of criterions comprises one or moreof: the attribute values are resulting from a data standardizationalgorithm, the attribute representing a primary (PK) or foreign key (FK)of the dataset, the attribute values have a predefined data class, theattribute does not have similar characteristics as another attribute ofthe dataset, wherein values of the other attribute are standardized, thenumber of different formats of the attribute is lower than a number offormats threshold, the average length of the values of the attribute islower than a length threshold, the average number of words of theattribute is lower than a number of words threshold, the fraction of thedistinct values of the attribute is lower than fraction threshold. Thisembodiment may provide as much conditions as possible that may eradicatethe need for the column to be considered for data standardization andthe need for data standardization can return a very low number.

According to one embodiment, the method is performed during dataprofiling of the dataset. For example, the metadata may further comprisethe profiling results that may be obtained while performing the dataprofiling e.g., the method may be performed as soon as the profilingresults are available and before performing ETL processes. This may beadvantageous as the metadata may further include the data providingresults that result from profiling the dataset. The profiling results inaddition to the metadata may provide an enriched source of informationon the columns such that the probability of fulfilling the conditionsdescribed above may be increased compared to a reduced source ofinformation. Thus, this may further reduce the number of times thestandardization score algorithm is executed. The need of standardizationmay be a potential data quality problem that needs to be resolved asearlier as possible in an event involving ETL processes e.g. at the dataprofiling stage before integrating the data. This earlierstandardization may provide a clean data warehouse sample that can beused for performing an accurate data analysis.

According to one embodiment, the standardization score algorithmcomprises an algorithm for calculating similarity between attributevalues and calculating the score based on the similarities. For example,the attribute values may be of string type. The attribute values of thedataset may be distinct values that are determined by (or are the outputof) a given deduplication algorithm. However, since such deduplicationalgorithms are incapable of finding potential duplicates efficiently,using the standardization score algorithm may overcome that issue byfinding potential duplicates in a time efficient manner.

According to one embodiment, the standardization score algorithm isconfigured for: converting each attribute value (the attribute valuebeing a string) to a respective set of bigrams; determining all bigramspresent in the attribute values; representing bigrams present in theattribute values as bits (or binary numbers), resulting in a bitmaprepresenting the presence of bigrams in the attribute values; groupingattribute values using bitwise operations on the bitmap, wherein eachgroup comprises attribute values that are determined based on pairwisebigram-based similarity scores, the pairwise bigram-based similarityscore reflecting the number of common bigrams between two attributevalues; calculating the data standardization score using the number ofgroups. The pairwise bigram-based similarity score may be determined foreach pair of attribute values. The pairwise bigram-based similarityscore of a pair may for example comprise the number of common bigramsbetween the attribute values of the pair. A pair of attribute values maybe included in a group if the pairwise bigram-based similarity score ishigher than a predefined score threshold. For example, the scorethreshold may be a predefined minimum number of common bigrams.

According to one embodiment, the grouping of attribute valuescomprising: selecting a first bit position in the bit map and checkingwhich set of attribute values have the bigram corresponding to the firstbit position; grouping attribute values having the bigram correspondingto the first bit position in one or more groups, wherein each groupcomprises attribute values of the set of attribute values that aredetermined based on pairwise bigram-based similarity scores, thepairwise bigram-based similarity score reflecting how many bigrams twoattribute values have in common; in case the number of groups identifiedis smaller than a predefined threshold, repeating the selecting andgrouping steps using a second bit position and non-clustered attributevalues until the number of groups is higher than the predefinedthreshold or until all bits are processed or until all attribute valuesare grouped; calculating the data standardization score using the numberof groups. The standardization score algorithm may further be configuredfor removing duplicates from data that are identified e.g. in theprofiling process.

This may be advantageous as it may find in a long list of values, groupof values that are likely to be a different spelling of the same value.This embodiment works at the values level and allows one to identify the2 single values which are (strictly speaking) different strings (e.g.,different spellings of the same value). This is by contrast toconventional methods that cannot detect that 2 values having similarwords in a different order are likely to be the same: ex: Rob Alice vs.Alice Robert.

This embodiment may efficiently cluster the similar data together andalso efficiently compare them to find potential duplicates within themwithout the knowledge of the domain of the data.

This may provide an efficient method for finding potential duplicatevalues, in particular in large set of unclassified data.

Other advantages of this embodiment may be the following advantages:

Comparing bits is much faster than comparing strings. So this may makethe algorithm much more scalable.

The present method may not need prior sorting of the data beforecomparison which most of the matching algorithms do. Sorting a hugedataset is a time consuming overhead that was avoided.

The present method may not need to do classification or standardizationof the data to find potential duplicates.

There is a formula to generate a matching threshold while comparing thebits of two attribute values. The formula as described below is thefollowing (S1∩S2)/(S1US2) (the ratio of (S1∩S2) and (S1US2)) where S1are the bits that are set to 1 a first attribute value and S2 are thebits that are set to 1 for a second attribute value.

Converting the data to phonetic (like Metaphone) during preprocessingmay reduce drastically the number of bigrams generated and thus make theclustering much faster. The more stricter algorithms, post clusteringcan very well make up for some of the false positives that could havebeen generated due to this liberality.

This method may particularly be advantageous to find existence ofduplicates in the cleansed Master Data.

For example, if there is a spelling mistake in a bigram then the presentmethod may be able to cluster it together such as for example wordsPARIS and PAERIS. However, the present method may not group together theterms PARIS and ARIZONA even of presence of the bigram AR, because thepresent method may take care of this by doing left shift multiple times.So if PARIS and PAERIS do not match in first Bigram comparison, it mayget another chance.

According to one embodiment, the grouping of the attribute valuescomprising: performing a bitwise operation for each pair of attributesvalues of the attribute values for determining the number of commonbigrams between the pair of attribute values, recursively building eachgroup by including a first pair of attribute values in the group andincluding each further pair of attribute values that shares at least oneattribute value with previously included pairs of the group. Forexample, if val1 is linked to val2 to form the first pair, then putrecursively val1, val2 and all values which are directly or indirectlylinked to val1 or val2 in the same group. For example, if there arepairs like (val1, val2), (val2, val3) and (val3, val6), the first pair(val1, val2) may be included in the group, and since val2 is linked toval3 in the second pair, the second pair may also be included in thegroup (e.g., only val3 will be added to the group because val2 alreadyincluded in the group and no need to duplicate it), then it may bedetermined that val6 is linked to val3 in the third pair, and thus thethird pair may also be included in the group (e.g., only val6 will beadded to the group because val3 already included in the group and noneed to duplicate it) and the same may be done for other groups.

According to one embodiment, the method further comprises: removing fromthe bitmap bigrams which are present in all attribute values. Thisembodiment may reduce the size of the bitmap by rejecting bigrams thatmay not be useful for the potential duplicates identification. And maythus reduce the processing resources required for processing the bitmap.

According to one embodiment, the method further comprises: for eachgroup of attribute values of the groups that result from the groupingstep:

For each pair of attribute values of the group:

splitting a first attribute value of the pair into first words and asecond attribute value of the pair into second words, the first wordsbeing constituent parts of the first attribute value, the second wordsbeing constituent parts of the second attribute value;

determining all bigrams present in each first and second words;

determining pairs of first and second words having a number of commonbigrams higher than a predefined threshold;

determining a word level character-based similarity score for eachdetermined pair of words;

combining the determined word level character-based similarity scores todetermine a character-based similarity score for the pair;

selecting the pairs of attribute values whose character-based similarityscore is higher than a predefined similarity threshold;

clustering pairs of the selected pairs that share one attribute valueinto a respective cluster, thereby resulting in one or more clusters.(e.g., the clustering of the pairs may be performed recursively asdescribed herein).

For example, the first (second) words may be words that are separated bya separator, such as a space or coma, etc. This embodiment may furtherincrease the accuracy of the present method for identification of thepotential duplicates.

According to one embodiment, the pairwise bigram-based similarity scoreis determined only for pairs of attribute values having a difference inlength that is smaller than a predefined maximum difference. Forexample, the two attribute values of the pair may have similar or thesame length in order to be compared. This may be advantageous as theattribute values (e.g., strings) may be broken in groups based on sizeso that only similar sized strings may be compared. This may not groupstrings like “Paris” and “Arizona tourist office center” togetherbecause of size even though they have same bigram “ar”. Lengths could beoverlapping (0-15/10-25/20-35/ . . . ). This may save processing timethat would otherwise be required by comparing those different strings.

According to one embodiment, the pairwise bigram-based similarity scorebeing determined using the number of bit pairs having same bits that twoattribute values have. For example, the pairwise bigram-based similarityscore may be the ration of the determined number of bit pairs divided bythe number of bit pairs that have different bits of the two attributevalues. This may provide an accurate scoring method based on accuratelyidentified duplicates.

According to one embodiment, for each group of attribute values:determining for each pair of attribute values a character-basedsimilarity score using a similarity algorithm; selecting the pairs ofattribute values whose character-based similarity score is higher than apredefined similarity threshold; clustering pairs of the selected pairsthat share one attribute value into a respective cluster, therebyresulting in one or more clusters. Using further similarity checkalgorithm on top of the standardization score algorithm may furtherincrease the accuracy of the present method and may provide cleansedgroups. Thus, the resulting score may be more accurate.

According to one embodiment, the determining of the character-basedsimilarity score for the each pair comprising: comparing each word of afirst attribute value of the each pair with each word of a secondattribute value of the each pair. For example, for each comparison aword-level character-based similarity score may be determined orcalculated and the character-based similarity score for the each pairmay be a combination (e.g., the sum) of the word-level character-basedsimilarity scores.

According to one embodiment, calculating the data standardization scoreusing the number of groups comprises: determining the number ofattribute values that are comprised in the clusters, wherein the datastandardization score is the ratio of the determined attribute values tothe number of attribute values in the dataset. This may provide areliable estimation of the score in particular for large datasets. Thisis by contrast to a scoring based on absolute number of duplicates thatare found.

According to one embodiment, the similarity algorithm comprises at leastone of edit distance and Levenshtein edit distance algorithms. Thisembodiment may be advantageous as it may seamlessly be integrated withexisting systems.

According to one embodiment, the length of the bitmap is determined bythe number of different bigrams occurring at least once in the dataset.This may provide reliable decomposition of the attribute values in orderto find duplicates. This is by contrast to determining the length of thebigram as the total number of theoretical possible bigrams.

According to one embodiment, the bigram is a sequence of two or moreadjacent elements or characters of the attribute value. The presentmethod may be applied using N-grams as described herein with the2-grams, wherein the N-gram is a sequence of N adjacent characters of anattribute value.

According to one embodiment, representing bigrams present in theattribute values as binary numbers further comprises removing from thebitmap all columns of bits where the value of the bit is the same forall rows of the bitmap. This may speed up the processing of the presentmethod.

According to one embodiment, the attribute is a string type.

FIG. 1 represents a general computerized system, suited for implementingmethod steps as involved in the disclosure.

It will be appreciated that the methods described herein are at leastpartly non-interactive, and automated by way of computerized systems,such as servers or embedded systems. In exemplary embodiments though,the methods described herein can be implemented in a (partly)interactive system. These methods can further be implemented in software112, 122 (including firmware 122), hardware (processor) 105, or acombination thereof. In exemplary embodiments, the methods describedherein are implemented in software, as an executable program, and isexecuted by a special or general-purpose digital computer, such as apersonal computer, workstation, minicomputer, or mainframe computer. Themost general system 100 therefore includes a general-purpose computer101.

In exemplary embodiments, in terms of hardware architecture, as shown inFIG. 1, the computer 101 includes a processor 105, memory (main memory)110 coupled to a memory controller 115, and one or more input and/oroutput (I/O) devices (or peripherals) 10, 145 that are communicativelycoupled via a local input/output controller 135. The input/outputcontroller 135 can be, but is not limited to, one or more buses or otherwired or wireless connections, as is known in the art. The input/outputcontroller 135 may have additional elements, which are omitted forsimplicity, such as controllers, buffers (caches), drivers, repeaters,and receivers, to enable communications. Further, the local interfacemay include address, control, and/or data connections to enableappropriate communications among the aforementioned components. Asdescribed herein the I/O devices 10, 145 may generally include anygeneralized cryptographic card or smart card known in the art.

The processor 105 is a hardware device for executing software,particularly that stored in memory 110. The processor 105 can be anycustom made or commercially available processor, a central processingunit (CPU), an auxiliary processor among several processors associatedwith the computer 101, a semiconductor based microprocessor (in the formof a microchip or chip set), a macroprocessor, or generally any devicefor executing software instructions.

The memory 110 can include any one or combination of volatile memoryelements (e.g., random access memory (RAM, such as DRAM, SRAM, SDRAM,etc.)) and nonvolatile memory elements (e.g., ROM, erasable programmableread only memory (EPROM), electronically erasable programmable read onlymemory (EEPROM), programmable read only memory (PROM). Note that thememory 110 can have a distributed architecture, where various componentsare situated remote from one another, but can be accessed by theprocessor 105.

The software in memory 110 may include one or more separate programs,each of which comprises an ordered listing of executable instructionsfor implementing logical functions, notably functions involved inembodiments of this invention. In the example of FIG. 1, software in thememory 110 includes instructions 112 (e.g., instructions to managedatabases, such as a database management system). The memory 110 mayfurther comprise a query optimizer. The query optimizer may compriseinstructions (e.g., software instructions that when executed may providea query execution plan for executing a given query).

The software in memory 110 shall also typically include a suitableoperating system (OS) 111. The OS 111 essentially controls the executionof other computer programs, such as possibly software 112 forimplementing methods as described herein.

The methods described herein may be in the form of a source program 112,executable program 112 (object code), script, or any other entitycomprising a set of instructions 112 to be performed. When a sourceprogram, then the program needs to be translated via a compiler,assembler, interpreter, or the like, which may or may not be includedwithin the memory 110, so as to operate properly in connection with theOS 111. Furthermore, the methods can be written as an object orientedprogramming language, which has classes of data and methods, or aprocedure programming language, which has routines, subroutines, and/orfunctions.

Software 112 may, for example, comprise a data standardization scorealgorithm 163 for finding potential duplicates in attribute values of adataset. The data standardization score algorithm 163 may be configuredfor calculating a data standardization score according to the potentialduplicates found. The calculated data standardization score reflectswhether data quality of attribute values would increase if astandardization rule is applied to the attribute values.

In exemplary embodiments, a conventional keyboard 150 and mouse 155 canbe coupled to the input/output controller 135. Other output devices suchas the I/O devices 145 may include input devices, for example, but notlimited to, a printer, a scanner, microphone, and the like. Finally, theI/O devices 10, 145 may further include devices that communicate bothinputs and outputs, for instance, but not limited to, a networkinterface card (NIC) or modulator/demodulator (for accessing otherfiles, devices, systems, or a network), a radio frequency (RF) or othertransceiver, a telephonic interface, a bridge, a router, and the like.The I/O devices 10, 145 can be any generalized cryptographic card orsmart card known in the art. The system 100 can further include adisplay controller 125 coupled to a display 130. In exemplaryembodiments, the system 100 can further include a network interface forcoupling to a network 165. The network 165 can be an IP-based networkfor communication between the computer 101 and any external server,client and the like via a broadband connection. The network 165transmits and receives data between the computer 101 and externalsystems 30, which can be involved to perform part or all of the steps ofthe methods discussed herein. In exemplary embodiments, network 165 canbe a managed IP network administered by a service provider. The network165 may be implemented in a wireless fashion, e.g., using wirelessprotocols and technologies, such as WiFi, WiMax, etc. The network 165can also be a packet-switched network such as a local area network, widearea network, metropolitan area network, Internet network, or othersimilar type of network environment. The network 165 may be a fixedwireless network, a wireless local area network (LAN), a wireless widearea network (WAN) a personal area network (PAN), a virtual privatenetwork (VPN), intranet or other suitable network system and includesequipment for receiving and transmitting signals.

If the computer 101 is a PC, workstation, intelligent device or thelike, the software in the memory 110 may further include a basic inputoutput system (BIOS) 122. The BIOS is a set of essential softwareroutines that initialize and test hardware at startup, start the OS 111,and support the transfer of data among the hardware devices. The BIOS isstored in ROM so that the BIOS can be executed when the computer 101 isactivated.

When the computer 101 is in operation, the processor 105 is configuredto execute software 112 stored within the memory 110, to communicatedata to and from the memory 110, and to generally control operations ofthe computer 101 pursuant to the software. The methods described hereinand the OS 111, in whole or in part, but typically the latter, are readby the processor 105, possibly buffered within the processor 105, andthen executed.

When the systems and methods described herein are implemented insoftware 112, as is shown in FIG. 1, the methods can be stored on anycomputer readable medium, such as storage 120, for use by or inconnection with any computer related system or method. The storage 120may comprise a disk storage such as HDD storage.

The storage 120 may comprise at least one dataset (or data table) 127.For example, the software 112 may receive (automatically or uponrequest) as input the dataset 127, or may download the dataset 127 fromstorage 120 or memory 110.

The dataset 127 may comprise one or more columns (e.g., 167-169),wherein each column is represented by a respective attribute “Att1” and“Att2”. The rows or records of the dataset 127 may comprise values ofthe attributes (attributes and columns are used interchangeably). Theattributes 167-169 may for example be a string type value.

The term “dataset” or data table as used herein refers to a collectionof data that may be presented in tabular form. Each column in the datatable may represent a particular variable or attribute. Each row in thedata table may represent a given member, record or entry of the datatable. In another example, the dataset may have a hierarchical structurelike JSON or XML format. In another example, the dataset may berepresented as a graph, or may be represented in a triplet format likeRDD or may have a structured format that may be accessed and used inaccordance with the present method.

The one or more attributes 167-169 may be described using metadata 161that may for example be stored in storage 120.

FIG. 2 is a flowchart of a method for determining a data standardizationscore for an attribute 167 of a dataset 127. For example, the presentmethod may be applied on each attribute of the dataset 127. In anotherexample, the present method may be applied for an attribute 167 of thedataset 127, wherein the attribute 167 is for example user defined. Inanother example, the attribute 167 to be processed may automatically bechosen (e.g., randomly chosen).

In step 201, at least part of attribute values of the attribute 167 ofthe dataset 127 may be determined or defined in order to be processed inaccordance with the present method. For example, the at least part ofattribute values of the attribute of the dataset 127 may be user defined(e.g., a user input may be received indicating that at least part ofattribute values of the attribute of the dataset 127). In anotherexample, the at least part of attribute values of the attribute of thedataset 127 may be randomly selected or sampled from the dataset 127(e.g., a predefined number of records or rows of the dataset 127 may beselected from the dataset 127). The at least part of attribute values ofthe attribute 167 of the dataset 127 may be received (e.g., from a useror may automatically be accessed e.g. as soon as stored on the storage120). The term “user” may refer to an entity e.g., an individual, acomputer, or an application executing on a computer.

In another example, all attribute values of the attribute of the dataset127 may be processed. This may enable an accurate scoring of theattribute compared to a score based on a sample of the dataset 127.

In one example, the attribute values of the attribute 167 that are inthe dataset 127 may be the results of a deduplication algorithm that isapplied on the values of the attribute 167 (e.g., the attribute valuesof the attribute 167 are distinct values). The deduplication algorithmmay be configured to remove duplicates from the dataset 127 based onvalues of the attribute 167. This may be advantageous as it may speed upthe present method as it may run on a clean sample. Although two valuesmay be classified as distinct values by the deduplication algorithm, thetwo values may not be “true” distinct values. For example, the twovalues may be duplicates but may have spelling errors or may have anorder in their constituent parts which is different and thus thededuplication algorithm may not detect them. The two values may beidentified as duplicates or potential duplicates by the present methodas described herein.

The metadata 161 for the attribute 167 may be used to determine whether(inquiry 203) an indication to carry or not to carry out standardizationis available for the at least part of attribute values of the attribute167 of the dataset 127.

The metadata 161 of the attribute 167 may for example comprise or storesvalues of parameters or variables that describe the attribute 167. Forexample, the metadata 161 may comprise a Boolean that indicates whetherthis column or attribute 167 can be null; the type of the attribute 167;Boolean indicating whether this column or attribute 167 is a primarykey. The metadata 161 may further contain the data class of theattribute 167, references to external terms or tags linked to theattribute 167, information indicating where the values of the attribute167 come from (e.g., data lineage information), indication if astandardization or cleansing already occurred on the values of theattributes 167 before being stored in the dataset 127 and statisticalinformation on the values of the attribute 167 and theircharacteristics.

For example, the metadata 161 may be read to verify one or morepredefined conditions or criteria on the attribute values of theattribute 167 (e.g., that are to be satisfied by the attribute values ofthe attribute 167 in order to carry the standardization). For example,each condition of the predefined conditions indicates the behavior ofthe attribute values that would (clearly) indicate to carry or not tocarry out standardization for the attribute 167.

A condition or criteria of the multiple conditions may be an expressionthat may for example be created using an operator. The expression mayform a so-called relational expression or a condition. The expressionmay consist of a data item (which may for example be one of thevariables or parameters of the metadata 161), an operator or negationoperator, and a value. The term “operator,” as used herein, refers to anoperator that tests values or range of values against a data item. Theoperator may for example comprise a relational operator that tests ordefines a relation or condition between two entities including. Thesemay include equality (e.g., isPKcolumn=True) where isPKcolum is one ofthe variables of the metadata 161.

In one example, a condition of the predefined conditions may check ifthe attribute 167 represents a PK or FK column. If the attribute 167represents a PK or FK column then this is an indication to not carry outstandardization for the attribute 167. This is because columns orattributes that are part of a PK/FK relationships are no candidate forstandardization due to the fact that they are building a good PK/FKrelationship indicates that they are already in the expected format.Defined PK/FK relationships and candidate relationships can be retrievedfrom a metadata catalogue that comprises the metadata or from the dataprofiling results of the attribute 167. FIG. 4 provides further detailson the predefined conditions.

In another example, the metadata of the attribute 167 may comprise theindication to carry or not to carry out standardization e.g. in a formof a flag that is set to a value that indicates to carry or not to carryout standardization for the attribute 167.

In response to finding the indication, a respective value may be set instep 205 for the data standardization score. For example, the datastandardization score may be set to value 1.

In response to not finding the indication, the data standardizationscore algorithm 163 may be run in step 207 on the at least part ofattribute values of the dataset, which may result in the standardizationscore.

In case of multiple predefined conditions (e.g., two conditions: cond1,cond2), the indication may be verified for each condition (e.g.,sequentially) and only if all conditions are not verified then step 207may be performed. For example, if cond1 is not verified in that thestandardization may not be performed based on the result of verificationof cond1, the second condition, cond2, may be verified and only if alsothe second condition, cond2, is not verified in that the standardizationmay not be performed based on the result of verification of cond2 thenthe data standardization score algorithm 163 may be run.

In another example, a condition (e.g., cond2) may be the combination ofmultiple conditions (e.g., subcond1, subcond2). In this case, if atleast one of the sub-conditions, subcond1 and subcond2, is not verified,then the whole condition, cond2, is not verified. And if allsub-conditions, subcond1 and subcond2, are verified, then the wholecondition, cond2, is verified.

The standardization score that is calculated by the data standardizationscore algorithm 163 may depend on the number duplicates in attributevalues of the attribute 167. For example, the data standardization scoremay be the ratio of the duplicate attribute values of the attribute 167to the number of attribute values of the attribute 167 in the dataset127.

In step 209, the data standardization score value may be compared to apredefined criterion to determine whether data standardization is to beapplied on the attribute 167. For example, if the predefined criterionis fulfilled then the data standardization may be applied on all valuesof the attribute 167 of the dataset 127; otherwise the datastandardization may not be applied on the attribute 167 in the dataset127. The predefined criterion may comprise: the data standardizationscore value is higher than a threshold (e.g., 0.6).

The method of FIG. 2 may be performed during the data profiling stage ofthe dataset 127, wherein the metadata 161 may further comprise profilingresults of the dataset 127 which may be used in inquiry 203. The dataprofiling may indicate which records of the dataset 127 are potentiallyinteresting for performing the present method (e.g., indicating recordsinteresting for performing inquiry 203). The data profiling may furtheranalyze the data to retrieve information for each analyzed columns, suchas their inferred types, general statistics about the values itcontains, common formats, value distributions, etc.

FIG. 3 is a flowchart of a method for finding groups of potentialduplicates, which is performed by the scoring the standardizationalgorithm 163.

In step 301, each attribute value of the at least part of attributevalues of the attribute 167 of the dataset 127 may be converted to arespective set of bigrams. The bigrams may for example comprise asequence of two adjacent elements of the attribute value. For example,if the attribute value is “PATENT”, then the set of bigrams of theattribute value “PATENT” is “PA”, “AT”, “TE”, “EN” and “NT”.

In step 303, all bigrams present in the attribute values of theattribute 167 may be determined. For example, all bigrams may be storedin an array where each element of the array may comprise a bigram and anindication of the corresponding attribute value in the dataset 127.

In step 305, bigrams (e.g., as determined in step 303) present in theattribute values may be represented as binary numbers. This may resultin a bitmap representing the presence of bigrams in the attributevalues. The bitmap or bit array may be an array data structure thatcompactly stores bits.

The bitmap refers to a bit array (e.g., a two dimensional bit array) inwhich each set of bits, represents or corresponds to an item (e.g., anattribute value). For example, the bitmap may comprise a bit matrix tostore two dimensional arrays of 0 or 1 values. The bitmap may comprise mrows and n columns, where m refers to the number of attribute values andn refers to the number of bigrams. Each row in the bitmap may representa given attribute value (the row comprises the set of bits thatcorresponds to the given attribute value) and each column in the bitmapmay represent a bigram. Each cell of the bitmap has a value of 0 or 1.The value 0 (1) indicates that the attribute value that corresponds tothe cell does not comprise (does comprise) the bigram that correspondsto the cell. A 0 or 1 value in the bit matrix uses 1 bit. This datastructure may have the advantage of reducing the memory usage comparedto a normal data structure.

In order to access the bitmap, the present method may provide and/or usepredefined information indicating which bigram corresponds to whichposition in the bitmap (ex: bigram aa corresponds to the 1st bit, bigramab, the second, etc.) and/or which attribute value corresponds to whichposition in the bitmap.

For example, the bitmap may be created in step 305 and defined such thateach column of bits of the bitmap represents a respective bigram thathas been found or determined for one or more attribute values of theattribute 167. The column of bits comprises bits that are sets for eachattribute value of the attribute 167. The bitmap comprises a set of bitsfor each of the attribute values to be processed. The set of bitscomprises a number of bits that corresponds to the number of bigrams inall the attribute values to be processed (e.g., each bit in a set ofbits corresponds to a respective bigram).

For exemplification purpose, assuming that attribute values to beprocessed are “PATENT”, “ATE”, “APTENT”, the list of bigrams maycomprise “PA”, “AT”, “TE”, “EN”, “NT”, “AP” and “PT” the bitmap may havethe following structure:

TE AT PA EN NT AP PT 1 1 1 1 1 1 0 0 2 1 1 0 0 0 0 0 3 1 0 0 1 1 1 1

The first row indicates the bits that are associated with the attributevalue “PATENT”. The bits of the first row are set such that theyrepresent the content of the attribute value “PATENT”. Since theattribute value “PATENT” has the bigrams “PA”, “AT”, “TE”, “EN” and“NT”, the corresponding bits are set to 1, while the remaining bits thatcorrespond to the other bigrams “AP” and “PT” are set to 0 since none ofthem is contained in the attribute value “PATENT”.

The second row indicates the bits that are associated with the attributevalue “ATE”. The bits of the second row are set such that they representthe content of the attribute value “ATE”. Since the attribute value“ATE” has the bigrams “AT” and “TE”, the corresponding bits are set to1, while the remaining bits that correspond to the other bigrams “PA”,“EN”, “NT”, “AP” and “PT” are set to 0 since none of them is containedin the attribute value “ATE”.

The third row indicates the bits that are associated with the attributevalue “APTENT”. The bits of the third row are set such that theyrepresent the content of the attribute value “APTENT”. Since theattribute value “APTENT” has the bigrams, “TE”, “EN”, “NT”, “AP” and“PT” the corresponding bits are set to 1, while the remaining bits thatcorrespond to the other bigrams “PA” and “AT” are set to 0 since none ofthem is contained in the attribute value “APTENT”.

In step 307, the attribute values may be grouped, which may result in agiven number of groups. Each group of the resulting groups may compriseattribute values that have a number of common bigrams that is higherthan a predefined minimum number of common bigrams. The grouping may beperformed using bitwise operations of sets of bits of pairs of theattribute values to be processed.

Using the above example, among the attribute values “PATENT”, “ATE” and“APTENT” groups may be built. For that, three pairs of attribute valuesmay be considered namely: a first pair (“PATENT”, “ATE”), second pair(“PATENT”, “APTENT”) and third pair (“APTENT”, “ATE”).

For the first pair the bigrams that are in common are “TE” and “AT”which is a number of 2 common bigrams between attribute values “PATENT”and “ATE”.

For the second pair the bigrams that are in common are “TE”, “EN” and“NT” which is a number of 3 common bigrams between attribute values“PATENT” and “APTENT”.

For the third pair there is only one bigram “TE” that is in commonbetween attribute values “APTENT” and “ATE”.

In order to determine the number of bigrams (bits) which are present inboth attribute values of a pair, an AND bitwise operation between the 2sets of bits of the pair may be performed. In another example, thenumber of bigrams which are either present in both or absent in bothsets of bits of the pair may be counted using an NOT XOR bitwiseoperation may be used for defining the pairwise bigram-based similarityscore. In another example, bitwise operations may be used to calculatethe following similarity between two attribute values A and B:similarity(A, B)=nb_bits_set(A AND B)/nb_bits_set(A OR B). Where bitwiseoperation nb_bits_set(A AND B) counts the number of bits set to 1 inboth attribute values A and B and bitwise operation nb_bits_set(A OR B)counts the number of bits set to 1 in at least one attribute value A andB. A further example is shown with reference to FIGS. 5A-5D.

For example counting the common number of bits between 2 sets of bitsmay be performed by applying an AND operation between the 2 sets of bitsand counting the number of bits that are set to 1 in the result. Theseoperations may be done very efficiently by a CPU. This common number ofbits between two sets of bits may be indicative of the number of commonbigrams between the two sets of bits

For example, for the first pair, the NAND result between set of bits of“PATENT” and set of bits of “ATE” is 1100000 which indicates that 2pairs of bits have same values 1 in the first pair (or two bigrams arepresent in both attribute values of the first pair).

For the second pair, the AND result between set of bits of “PATENT” andset of bits of “APTENT” is 1001100 which indicates that 3 pairs of bitshave same values 1 in the second pair.

For the third pair, the AND result between set of bits of “ATE” and setof bits of “APTENT” is 10000000 which indicates that 1 pair of bits havesame values in the third pair.

In order to build a group of attribute values, a pairwise bigram-basedsimilarity score may be determined for each of the three pairs. Thepairwise bigram-based similarity score may for example equal to theration of the common bigrams for a given pair over all bigrams (e.g.,that are found in step 303). In this case, the first, second and thirdpairs may have the bigram-based similarity score of 2/7, 3/7 and 1/7respectively.

The calculated bigram-based similarity scores may be compared with apredefined threshold e.g. 0.25 to check whether they are higher than0.25. In this case, the first and second pairs would service thecondition and may then be grouped in one group since they share oneattribute value “PATENT”. The one group may comprise the attributevalues “PATENT”, “APTENT” and “ATE”.

However, if the predefined threshold is 0.35, then only the second pairwould service the condition and may then be used to build or define onegroup comprising the attribute values “PATENT” and “APTENT”.

In step 309, the data standardization score may be calculated using thegroups. For example, the data standardization score may be defined asthe ratio of the determined attribute values to the number of attributevalues in the dataset. Using the above example (with threshold 0.25),this ratio is 1 meaning that 100% of the attribute values of the datasetare grouped. By comparing this data standardization score to a scorethreshold it may be decided whether or not to standardize the attributevalues. Using the above example, if the score threshold is 30% then theattribute values PATENT”, “APTENT” and “ATE” may be standardized.

Before performing step 309, the groups defined in step 307 may furtherbe processed for checking or running on them other similarityalgorithms. For example, the group comprising attribute values PATENT”,“APTENT” and “ATE” may further be processed by calculating for thefirst, second and third pairs a respective character-based similarityscore using a similarity algorithm. The similarity algorithm may be atleast one of edit distance and Levenshtein edit distance algorithms. Iffor example the character-based similarity score for the first and thirdpairs are below a given character scoring threshold, then the group maybe redefined such that the attribute value “ATE” is excluded from thegroup of attribute values PATENT”, “APTENT” and “ATE”. The resultinggroup may comprise only PATENT” and “APTENT”.

FIG. 4 is a flowchart of a method for determining, based on the metadata161 for the attribute 167, whether an indication to carry or not tocarry out standardization is available for attribute values of thedataset (further detailing inquiry 203).

In step 401, metadata 161 and profiling results of the attribute orcolumn 167 may be fetched.

In inquiry 403, it may be determined if column 167 is alreadystandardized (in other words the column 167 is a result of datastandardization). Data columns that are themselves result from datastandardization process can be ignored from standardization process. Thecolumn names could be found in standardization dictionaries (e.g. thatis part of the metadata 161) and the column with those names could beignored. 0.0 can be returned for these. Thus, if column 167 is alreadystandardized step 417 may be performed; otherwise inquiry 405 may beperformed. In step 417 the score may be set to value 0 (e.g. value of 0is to indicate that no standardization is needed). In another example,column 167 may be considered as already standardized if column 167 hasbeen indirectly standardized. That is, column 167 is the result of anETL process that is applied on source data that is a standardized data.For that, the data lineage information may be used to follow the data totheir source and check if it has already been standardized.

In inquiry 405, it may be determined if column 167 is a Primary Key orForeign Key (e.g. either because column 167 is inferred as a good PK/FKor because it is part of an actual PK/FK relationship). Columns that arepart of a PK/FK relationships are no candidate for standardizationbecause the fact that they are building a good PK/FK relationshipindicates that they are already in the expected format. Defined PK/FKrelationships and candidate relationships can be retrieved from themetadata 161 or the data profiling results that may be part of themetadata 161. Thus, if column 167 is a Primary Key or Foreign Key step417 may be performed; otherwise inquiry 407 may be performed. In step417 the score may be set to value 0.

In inquiry 407, it may be determined if column 167 is classified as adata class for which no standardization is known to be necessary. Dataprofiling tools can classify the kind of data contained in a column. Forexample, Product Image, Identifier Columns (say True or False flags) donot require data standardization. So if the column contains these dataclass, they can be ignored from the perspective of data Standardization.0.0 can be returned as the score. Thus, if column 167 is classified asthat data class, step 417 may be performed; otherwise inquiry 409 may beperformed. In step 417 the score may be set to value 0.

In inquiry 409, it may be determined if column 167 is classified as adata class for which a standardization rule exists. If the detected dataclass of a column is known to have standardization rules applicable forit, return 1.0 as it is known that it can be standardized. Thus, ifcolumn 167 is classified as that data class step 417 may be performed;otherwise inquiry 411 may be performed. In step 417 the score may be setto value 1.

In inquiry 411, it may be determined if column 167 has similarcharacteristics (e.g. domain fingerprint) or common values with anothercolumn of the dataset 127 which has been standardized before. If theanalyzed column contains data showing the same characteristics asanother column which has been standardized before (although the exactclassification of the column could not be determined), the need forstandardization is the confidence that the 2 columns represent the samedomain. For instance if it is known that standardization was applicablefor the data in FIG. 6A, and the data of FIG. 6B seem to containinformation of same type, then the need for standardization of the dataof FIG. 6B is the confidence that those two columns representinformation of same type. Computation of fingerprints allowing to testwhether 2 sets of values share enough characteristics to indicate thatthey are likely to be of same domain may be used. Thus, if column 167has similar characteristics as another column of the dataset 127 step417 may be performed; otherwise inquiry 413 may be performed. In step417 the score may be a similarity score with the most similar column ofthe dataset 127 which has been standardized.

In inquiry 413, it may be determined if average value lengths, averagenumber of words and number of formats of the column are below respectivepredefined thresholds. If it is the case, step 417 may be performedotherwise the standardization score algorithm 163 may be run in step 415on the attribute values of the attribute or column 167. In step 417 thescore may be set to value 0. After executing step 415, step 417 may beperformed. In this case, step 417 may comprise setting the score to avalue which is the output or result of running the standardization scorealgorithm.

In other words, if none of the previous criteria (403-411) applied,check (inquiry 413) for the following conditions for the column 167:

Data class category is ‘text’,

The number of different formats is above a threshold t1,

The average length of the values is above t2,

The average number of words is above t3, and if

The cardinality (number of distinct values/total number of values) isabove t4.

Longer values or values containing a larger number of words or having alarge number of different formats and values are more likely to becandidate for standardization as values with a constant format or welldefined different values. Note that here more advanced criteria could beused, such as analyzing if there are obvious outliers in the frequencydistributions of the values, like some values having an unusually lowfrequency compared to the other value. If it is the case, a deeperanalysis (163) may be run on this column that will search for nearlyduplicates. In other words, in case the general statistics about thecolumn 167 (data type, number of formats, distribution of value lengths,number of words, value distributions, etc. which may be part of metadata161) indicate that the column contains data which could potentially havea standardization problem because of their nature, a deeper analysisusing standardization algorithm 163 in these columns whose purpose is toget a good guess of the % of values in that column which have one ormore potential related values that could represent the same entityalthough the values are not exactly the same.

Steps 401-417 may be performed for each column or attribute of thedataset 127.

FIGS. 5A-5D illustrate steps for creating clusters of attribute valuesof the attribute 167 using a bitmap.

FIG. 5A depicts the list of attribute values 501.1 to 501.10 to beprocessed. Each of the attribute values 501.1-10 is associated with therespective set of bigrams 503.1-503.10 in the attribute value. Attributevalues 501.1 to 501.10 may be distinct values. The attribute values501.1-10 may be stored in a column 501 and corresponding sets of bigrams503.1-503.10 may be stored in column 503 of the table shown in FIG. 5A.

From the first step that is illustrated by FIG. 5A, the list of bigramsto be found in the attribute or column 501 is determined.

For each attribute value 501.1-10 a respective set of bits 502.1-10 maybe created as illustrated in FIG. 5B. In the bitmap 505, each pair ofbigrams and associated attribute values is represented by a bit. Bitvalue of 0 at a position represents that a particular bigram does notexist in the attribute value and 1 indicates that the particular bigramexists in the attribute value. This may end up having a large binarynumber associated with each attribute value. The 10 attribute values501.1-10 of the input have 63 different bigrams.

The length of the bitmap may be determined by the number of differentbigrams occurring at least once in the full dataset containing attributevalues 501. The length of the bitmap indicates the number of bits thathave a value of 1 in the bitmap (referred to as set of bits) associatedwith each attribute value (e.g., 63 bits).

In a following step as illustrated in FIG. 5C, a first bit position(e.g., most left bit of the bitmap) is used to find those bits thatreturn 1 and which are blocked together and compared. As illustrated inFIG. 5C, the first position may comprise bit position 509 whichrepresents the bigram “AL”. In this example, the 2nd and 10th rowsreturn 1 as values of bits 511 and 513 respectively.

The 2nd and 10th rows or attribute values 501.2 and 501.10 may becompared by comparing the bits within them. The bits of each attributevalues 501.2 and 501.10 are further separately listed in FIG. 5C.

For comparing the bits a similarity factor (e.g., the pairwisebigram-based similarity score) may be calculated for the two attributevalues 501.2 and 501.10. The similarity factor Sim(501.2, 501.10) may bedefined as follows Sim(501.2, 501.10)=(S1∩S2)/(S1US2), where S1 is thebits that are set to 1 for attribute value 501.2 and S2 is the bits thatare set to 1 for attribute value 501.10. The union (S1US2) refers to thepairs of bits 515.1-515.17, wherein each pair comprises bitscorresponding to the two attribute values 501.2 and 501.10 and comprisesat least one bit that is set to 1. In the example of FIG. 5C, there are17 pairs 515.1-17. The intersection (S1∩S2) refers to the pairs of bitsof 515.1-3 and 515.5-13, wherein each pair comprises two bits that areset to 1. In the example of FIG. 5C, there are 12 pairs.

Thus the similarity factor may be: Sim(501.2,501.10)=12/17=0.71.

Since the Similarity is above threshold (e.g. 0.25), the attributevalues 501.1 and 501.10 are clustered together as one cluster. One ofthem may become a cluster leader. Subsequent rows may be only comparedto this cluster leader.

Since sufficient clusters are not identified, the step described in FIG.5C may be repeated by doing a left shift to identify a second bitposition 520 which corresponds to bigram “AM” as shown in FIG. 5D. Rows3, 4, 6 and 9 (represented by set of bits 502.3, 502.4, 502.6 and 502.9)of the bitmap 505 or attribute values 501.3, 501.4, 501.6 and 501.9return 1 and are compared with each other by calculating the similarityfactor as defined above.

Sim (501.3, 501.4)=0.05 is below threshold 0.25 thus attribute values501.3 and 501.4 are not clustered together.

Sim (501.3, 501.6)=0.05 is below threshold 0.25 thus attribute values501.3 and 501.6 are not clustered together.

Sim (501.3, 501.9)=0.72 is higher than the threshold, thus attributevalues 501.3 and 501.9 are clustered together in cluster 523.

Sim (501.4, 501.6)=0.80 is higher than the threshold, thus attributevalues 501.4 and 501.6 are clustered together in cluster 521.

Sim (501.4, 501.9) and Sim (501.6, 501.9) are not performed as attributevalue 501.9 was already moved to cluster 523.

Sim (501.2, 501.3)=0 is below threshold 0.25 thus attribute values 501.2and 501.3 are not clustered together.

Sim (501.2, 501.4)=0.04 is below threshold 0.25 thus attribute values501.2 and 501.4 are not clustered together.

Since sufficient clusters (3) are identified, an exit may be performed.The following 3 clusters are identified, values within which may furtherbe compared with another string algorithm (UNCERT, etc.).

Cluster 519 has attribute values 501.2 and 501.10: CHEVEROLETTE MALIBUand CHEVROLET MALIBU.

Cluster 523 has attribute values 501.3 and 501.9: TOYOTA CAMRY andCAMREY TOYOTA.

Cluster 521 has attribute values 501.4 and 501.6: NISSAN MAXMA and NISANMAXIMA.

The exit condition may be quantified as follows: If over 25% of theattribute values 501.1-10 are covered in the clusters with more than 1record then the above exit may be performed (e.g., no repeating of stepof FIG. 5C is required).

The scoring may be performed as follows:

If 25% and above covered in clusters with more than 1 record

Score=1.0

Else

Score=Max(1.0, (% covered)*4/100).

For example, if the number of attribute values for which at least oneduplicate attribute value has been found is 25% of the total number ofattribute values, the result may be maximized to 1.0 because enoughnumber of attribute values with a potential duplicate value is foundwhich gives a high confidence that the standardization is needed for theattribute values.

The present method may have be advantageous as comparing 10 distinctvalues would have traditionally required n*(n+1)/2 comparisons. So from55 comparisons, the present method brought down the comparison to just5.

Another example for processing the bitmap 505 in order to findduplicates may comprise the following steps: splitting the bitmaps ingroups of 64 bigrams (which corresponds to 64 bits per attribute value)which can be easily processed by the CPU as 64 bits long numbers. Thus,for each attribute value N long number may be obtained. The attributevalues may be sorted by their first long numbers and compare withinsorted result each attribute value with only the next M values followingit (e.g., M=10 values) in order to find pairs of duplicate values. Thisoperation may be repeated by sorting by the 2nd long number of eachattribute value and determine if in that order new pairs of similar orduplicate values can be found, etc.

In another example, a method for determining a data standardizationscore for an attribute is provided. The method comprises: providingpredefined rules for determining based on metadata and characteristicsof an attribute, whether data quality of attribute values would increaseif a standardization rule is applied to the attribute values; providinga data standardization score algorithm and a criterion to compare acalculated data standardization score to, the calculated datastandardization score reflecting whether data quality of attributevalues would increase if a standardization rule is applied to theattribute values; receiving a set of attribute values for an attributeand metadata/characteristics for the attribute; (e.g., characteristicsmay be determined for the attribute); determining, based onmetadata/characteristics for the attribute, whether a clear indicationto carry or not to carry out standardization is available; in responseto finding the clear indication, setting a respective value for the datastandardization score; in response to not finding the clear indication,running the data standardization score algorithm on the attributevalues; comparing the data standardization score value to the criterionto determine whether data standardization is to be applied on theattribute. An example approach to calculate the data standardizationscore may comprise: converting each attribute value (string) to arespective set of bigrams; determining all bigrams present in theattribute values; representing bigrams present in the attribute valuesas binary numbers; determining similarity scores of attribute valuespairs based on how many bigrams the attribute value pairs share, thedetermining being done by bit operations on the binary numbersrepresenting the attribute values; clustering the attribute values basedon the similarity score.

The present invention may be a system, a method, and/or a computerprogram product. The computer program product may include a computerreadable storage medium (or media) having computer readable programinstructions thereon for causing a processor to carry out aspects of thepresent invention.

The computer readable storage medium can be a tangible device that canretain and store instructions for use by an instruction executiondevice. The computer readable storage medium may be, for example, but isnot limited to, an electronic storage device, a magnetic storage device,an optical storage device, an electromagnetic storage device, asemiconductor storage device, or any suitable combination of theforegoing. A non-exhaustive list of more specific examples of thecomputer readable storage medium includes the following: a portablecomputer diskette, a hard disk, a random access memory (RAM), aread-only memory (ROM), an erasable programmable read-only memory (EPROMor Flash memory), a static random access memory (SRAM), a portablecompact disc read-only memory (CD-ROM), a digital versatile disk (DVD),a memory stick, a floppy disk, a mechanically encoded device such aspunch-cards or raised structures in a groove having instructionsrecorded thereon, and any suitable combination of the foregoing. Acomputer readable storage medium, as used herein, is not to be construedas being transitory signals per se, such as radio waves or other freelypropagating electromagnetic waves, electromagnetic waves propagatingthrough a waveguide or other transmission media (e.g., light pulsespassing through a fiber-optic cable), or electrical signals transmittedthrough a wire.

Computer readable program instructions described herein can bedownloaded to respective computing/processing devices from a computerreadable storage medium or to an external computer or external storagedevice via a network, for example, the Internet, a local area network, awide area network and/or a wireless network. The network may comprisecopper transmission cables, optical transmission fibers, wirelesstransmission, routers, firewalls, switches, gateway computers and/oredge servers. A network adapter card or network interface in eachcomputing/processing device receives computer readable programinstructions from the network and forwards the computer readable programinstructions for storage in a computer readable storage medium withinthe respective computing/processing device.

Computer readable program instructions for carrying out operations ofthe present invention may be assembler instructions,instruction-set-architecture (ISA) instructions, machine instructions,machine dependent instructions, microcode, firmware instructions,state-setting data, or either source code or object code written in anycombination of one or more programming languages, including an objectoriented programming language such as Smalltalk, C++ or the like, andconventional procedural programming languages, such as the “C”programming language or similar programming languages. The computerreadable program instructions may execute entirely on the user'scomputer, partly on the user's computer, as a stand-alone softwarepackage, partly on the user's computer and partly on a remote computeror entirely on the remote computer or server. In the latter scenario,the remote computer may be connected to the user's computer through anytype of network, including a local area network (LAN) or a wide areanetwork (WAN), or the connection may be made to an external computer(for example, through the Internet using an Internet Service Provider).In some embodiments, electronic circuitry including, for example,programmable logic circuitry, field-programmable gate arrays (FPGA), orprogrammable logic arrays (PLA) may execute the computer readableprogram instructions by utilizing state information of the computerreadable program instructions to personalize the electronic circuitry,in order to perform aspects of the present invention.

Aspects of the present invention are described herein with reference toflowchart illustrations and/or block diagrams of methods, apparatus(systems), and computer program products according to embodiments of theinvention. It will be understood that each block of the flowchartillustrations and/or block diagrams, and combinations of blocks in theflowchart illustrations and/or block diagrams, can be implemented bycomputer readable program instructions.

These computer readable program instructions may be provided to aprocessor of a general purpose computer, special purpose computer, orother programmable data processing apparatus to produce a machine, suchthat the instructions, which execute via the processor of the computeror other programmable data processing apparatus, create means forimplementing the functions/acts specified in the flowchart and/or blockdiagram block or blocks. These computer readable program instructionsmay also be stored in a computer readable storage medium that can directa computer, a programmable data processing apparatus, and/or otherdevices to function in a particular manner, such that the computerreadable storage medium having instructions stored therein comprises anarticle of manufacture including instructions which implement aspects ofthe function/act specified in the flowchart and/or block diagram blockor blocks.

The computer readable program instructions may also be loaded onto acomputer, other programmable data processing apparatus, or other deviceto cause a series of operational steps to be performed on the computer,other programmable apparatus or other device to produce a computerimplemented process, such that the instructions which execute on thecomputer, other programmable apparatus, or other device implement thefunctions/acts specified in the flowchart and/or block diagram block orblocks.

The flowchart and block diagrams in the Figures illustrate thearchitecture, functionality, and operation of possible implementationsof systems, methods, and computer program products according to variousembodiments of the present invention. In this regard, each block in theflowchart or block diagrams may represent a module, segment, or portionof instructions, which comprises one or more executable instructions forimplementing the specified logical function(s). In some alternativeimplementations, the functions noted in the block may occur out of theorder noted in the figures. For example, two blocks shown in successionmay, in fact, be executed substantially concurrently, or the blocks maysometimes be executed in the reverse order, depending upon thefunctionality involved. It will also be noted that each block of theblock diagrams and/or flowchart illustration, and combinations of blocksin the block diagrams and/or flowchart illustration, can be implementedby special purpose hardware-based systems that perform the specifiedfunctions or acts or carry out combinations of special purpose hardwareand computer instructions.

The descriptions of the various embodiments of the present inventionhave been presented for purposes of illustration, but are not intendedto be exhaustive or limited to the embodiments disclosed. Manymodifications and variations will be apparent to those of ordinary skillin the art without departing from the scope and spirit of the describedembodiments. The terminology used herein was chosen to best explain theprinciples of the embodiments, the practical application or technicalimprovement over technologies found in the marketplace, or to enableothers of ordinary skill in the art to understand the embodimentsdisclosed herein.

1-18. (canceled)
 19. A computer program product for determining a datastandardization score for an attribute of a dataset, the computerprogram product comprising a computer readable storage medium havingprogram code embodied therewith, the program code comprising theprogramming instructions for: providing attribute metadata descriptiveof the attribute; providing a data standardization score algorithm forfinding potential duplicates in attribute values and calculating a datastandardization score accordingly, the calculated data standardizationscore reflecting whether data quality of attribute values would increaseif a standardization rule is applied to the attribute values;determining, based on the metadata for the attribute, whether anindication to carry or not to carry out standardization is available forat least part of attribute values of the dataset; in response to findingthe indication, setting a respective value for the data standardizationscore; in response to not finding the indication, running the datastandardization score algorithm on the at least part of attribute valuesof the dataset; and comparing the data standardization score value to apredefined criterion to determine whether data standardization is to beapplied on the attribute.
 20. A system, comprising: a memory unit forstoring a computer program for determining a data standardization scorefor an attribute of a dataset; and a processor coupled to the memoryunit, wherein the processor is configured to execute the programinstructions of the computer program comprising: providing attributemetadata descriptive of the attribute; providing a data standardizationscore algorithm for finding potential duplicates in attribute values andcalculating a data standardization score accordingly, the calculateddata standardization score reflecting whether data quality of attributevalues would increase if a standardization rule is applied to theattribute values; determining, based on the metadata for the attribute,whether an indication to carry or not to carry out standardization isavailable for at least part of attribute values of the dataset; inresponse to finding the indication, setting a respective value for thedata standardization score; in response to not finding the indication,running the data standardization score algorithm on the at least part ofattribute values of the dataset; and comparing the data standardizationscore value to a predefined criterion to determine whether datastandardization is to be applied on the attribute.
 21. The computerprogram product as recited in claim 19, wherein the attribute values aredistinct values of the attribute that are obtained by a deduplicationalgorithm.
 22. The computer program product as recited in claim 19,wherein the attribute values are all attribute values of the attributein the data set.
 23. The computer program product as recited in claim19, wherein the program code further comprises the programminginstructions for: providing a set of criterions, the determining, basedon the metadata for the attribute, whether the indication to carry ornot to carry out standardization is available comprising: checking eachof the criterions for the values of the attribute.
 24. The computerprogram product as recited in claim 23, wherein the set of criterionscomprises one or more of the following: the attribute values areresulting from a data standardization algorithm; the attribute valuesare resulting from an ETL process that is applied on source data thathas been standardized; the attribute is representing a primary orforeign key of the dataset; the attribute values have a predefined dataclass; the attribute has similar characteristics as another attribute ofthe dataset, wherein values of the other attribute are standardized; anumber of different formats of the attribute is above a number offormats threshold; an average length of the attribute values is above alength threshold; an average number of words of the attribute is above anumber of words threshold; and a fraction of the distinct values isabove a fraction threshold.
 25. The computer program product as recitedin claim 19, wherein the data standardization score algorithm comprisesan algorithm for calculating similarity between attribute values andcalculating the score based on the similarities.
 26. The computerprogram product as recited in claim 19, wherein the data standardizationscore algorithm is configured for: converting each attribute value to arespective set of bigrams; determining all bigrams present in theattribute values; representing bigrams present in the attribute valuesas bits, resulting in a bitmap representing the presence of the bigramsin the attribute values; grouping the attribute values using bitwiseoperations on the bitmap, wherein each group comprises attribute valuesof the attribute values having a predefined number of common bigrams;and calculating the data standardization score using a number of groups.27. The computer program product as recited in claim 26, wherein thegrouping of the attribute values comprises: performing a bitwiseoperation for each pair of attributes values of the attribute values,recursively building each group by including a first pair of attributevalues in the group and including each further pair of attribute valuesthat shares at least one attribute value with previously included pairsof the group.
 28. The computer program product as recited in claim 26,wherein the grouping of the attribute values comprises: selecting afirst bit position in the bitmap and checking which set of attributevalues have the bigram corresponding to the first bit position; groupingattribute values having the bigram corresponding to the first bitposition in one or more groups, wherein each group comprises attributevalues of the set of attribute values that are determined based onpairwise bigram-based similarity scores, the pairwise bigram-basedsimilarity score reflecting how many bigrams two attribute values havein common; and in case the number of groups is smaller than a predefinedthreshold, repeating the selecting and grouping steps using a second bitposition and non-grouped attribute values until the number of groups ishigher than the predefined threshold or until all bits are processed.29. The computer program product as recited in claim 28, wherein thepairwise bigram-based similarity score is determined using a number ofbit pairs having same bits of two attribute values.
 30. The system asrecited in claim 20, wherein the attribute values are distinct values ofthe attribute that are obtained by a deduplication algorithm.
 31. Thesystem as recited in claim 20, wherein the attribute values are allattribute values of the attribute in the data set.
 32. The system asrecited in claim 20, wherein the program instructions of the computerprogram further comprise: providing a set of criterions, thedetermining, based on the metadata for the attribute, whether theindication to carry or not to carry out standardization is availablecomprising: checking each of the criterions for the values of theattribute.
 33. The system as recited in claim 32, wherein the set ofcriterions comprises one or more of the following: the attribute valuesare resulting from a data standardization algorithm; the attributevalues are resulting from an ETL process that is applied on source datathat has been standardized; the attribute is representing a primary orforeign key of the dataset; the attribute values have a predefined dataclass; the attribute has similar characteristics as another attribute ofthe dataset, wherein values of the other attribute are standardized; anumber of different formats of the attribute is above a number offormats threshold; an average length of the attribute values is above alength threshold; an average number of words of the attribute is above anumber of words threshold; and a fraction of the distinct values isabove a fraction threshold.
 34. The system as recited in claim 20,wherein the data standardization score algorithm comprises an algorithmfor calculating similarity between attribute values and calculating thescore based on the similarities.
 35. The system as recited in claim 20,wherein the data standardization score algorithm is configured for:converting each attribute value to a respective set of bigrams;determining all bigrams present in the attribute values; representingbigrams present in the attribute values as bits, resulting in a bitmaprepresenting the presence of the bigrams in the attribute values;grouping the attribute values using bitwise operations on the bitmap,wherein each group comprises attribute values of the attribute valueshaving a predefined number of common bigrams; and calculating the datastandardization score using a number of groups.
 36. The system asrecited in claim 35, wherein the grouping of the attribute valuescomprises: performing a bitwise operation for each pair of attributesvalues of the attribute values, recursively building each group byincluding a first pair of attribute values in the group and includingeach further pair of attribute values that shares at least one attributevalue with previously included pairs of the group.
 37. The system asrecited in claim 35, wherein the grouping of the attribute valuescomprises: selecting a first bit position in the bitmap and checkingwhich set of attribute values have the bigram corresponding to the firstbit position; grouping attribute values having the bigram correspondingto the first bit position in one or more groups, wherein each groupcomprises attribute values of the set of attribute values that aredetermined based on pairwise bigram-based similarity scores, thepairwise bigram-based similarity score reflecting how many bigrams twoattribute values have in common; and in case the number of groups issmaller than a predefined threshold, repeating the selecting andgrouping steps using a second bit position and non-grouped attributevalues until the number of groups is higher than the predefinedthreshold or until all bits are processed.
 38. The system as recited inclaim 37, wherein the pairwise bigram-based similarity score isdetermined using a number of bit pairs having same bits of two attributevalues.